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Abstract. We introduce a new notion of rank for unitary representations of semisim- 
ple groups over a local field of characteristic zero. The theory is based on Kirillov's 
method of orbits for nilpotent groups over local fields. When the semisimple group is a 
classical group, we prove that the new theory is essentially equivalent to Howe's theory 
of iV-rank |Hwl| . |Lil| . Sea . Therefore, our results provide a systematic generalization 
of the notion of a small representation (in the sense of Howe) to exceptional groups. 
However, unlike previous works which used ad-hoc methods to study different types of 
classical groups (and some exceptional ones |Ws| . |LS|). our definition is simultaneously 
applicable to both classical and exceptional groups. The most important result of this 
paper is a general "purity" result for unitary representations, which demonstrates how 
similar partial results in the previous authors' works should be formulated and proved 
for an arbitrary semisimple group in the language of Kirillov's theory. The purity result 
is a crucial step towards studying small representations of exceptional groups. New 
results concerning small unitary representations of exceptional groups will be published 
in a forthcoming paper [Saj . 
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1. Introduction 

1.1. Background. The theory of iV-rank jHwlj . |Lilj . |Scaj provides an effective tool 
for studying "small" unitary representations of classical semisimple groups. It is based 
on analysing the restriction of a unitary representation to large commutative subgroups 
of such groups. The study of the notion of iV-rank began in [Hwl where the theory 
was developed for the metaplectic group. Roughly speaking, one can attach equivalence 
classes of bilinear symmetric forms to unitary representations, and these equivalence 
classes canonically correspond to orbits of the adjoint action of the Levi component of 
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the Siegel parabolic on its unipotent radical. Later on, the theory was extended by J. 
S. Li |Lilj and R.Scaramuzzi |Scaj to classical groups, and an intrinsic connection with 
the theta correspondence was also found in |Li2j . Representations which correspond to 
degenerate classes are called singular representations. They form a large class of non- 
tempered representations and therefore their study is essentially complementary to that 
of the tempered representations. It turns out that this class of representations is quite 
well-behaved. In fact Howe gave a construction of a large class of such representations 
Hw3j, and they are applied in construction of automorphic forms |Hw4j . [L"i3] . 

The lack of a theory of iV-rank which applies to the exceptional groups is in a sense 
a defect of this theory due to substantial interest in the small representations of these 
groups. Aside from some generalizations in the works of Loke and Savin |LSj and Sahi, 
most recently Weissman |Wsj provides an analogue for p-adic split simply-laced groups 
excluding Eg using the representation-theoretic analogue of a Fourier- Jacobi functor. 

Here we develop a theory of rank which can also be applied to (almost) all exceptional 
groups. The word "almost" will be explained more precisely in the remark after Propo- 
sition 13.1.11 Our theory relies on the structure of unipotent radicals of certain parabolic 
subgroups which are naturally expressible as a sequence of extensions by Heisenberg 
groups. In this paper these unipotent radicals are called H-tower groups (see Definition 

The main point of this work is to show that although the abelian nilradicals in terms of 
which rank was defined for classical groups are usually not available in exceptional cases, 
the maximal unipotent subgroups of all semisimple groups have a common feature which 
enables us to define a similar notion of rank, which is essentially equivalent to the older 
one for classical groups. This result, apart from being interesting itself, provides the 
main tool needed to study small representations of exceptional groups. One interesting 
application of this result is that it enables us to obtain sharp bounds on the behaviour 
of matrix coefficients of unitary representations of exceptional groups, a problem studied 
in |LZj . |Ohj and |LSj . Results along this line of research will be published elsewhere |5a| . 

It is also possible to generalize Theorem 4.2 of |Hwlj . which is the next step to- 
wards classification of small representations of exceptional groups. However, there does 
not seem to be any strong connection between the "exceptional theta correspondences" 
which result from the author's work and the ones that already exist in the literature (see 
[DS , |MSj or [GRSJ for instance). This makes the classification problem both harder 
and more interesting at the same time. This is a work still in its preliminary stages. 

This manuscript is organized as follows. Section 11.21 is devoted to the notation used 
throughout the article. In section 2 we recall some basics about the Heisenberg groups 
and the Weil representation. In section 3 we define a specific unipotent subgroup of a 
simple group with regard to which our rank is defined. The "rankable" representations 
of this unipotent group are defined and studied briefly in section 4. Section 5 is devoted 
to the statement and proof of the main theorems, which essentially assert that rank is a 
nontrivial invariant of a representation. Section 6 shows why our new theory generalizes 
the older one. 
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1.2. Notation. In this section we introduce some notation which will be used through- 
out this article. Throughout this paper, we will be working with a local field F which is 
either I, C or a finite extension of Q p , p ^ 2. Let F be the algebraic closure of F, and G 
be the F-points of an absolutely simple, simply connected linear algebraic group defined 
over F, and Gf be the group of F-rational points of G. We assume G is F-isotropic. 
By the Kneser-Tits conjecture, which holds in the case of local fields (see Proposition 
7.6 and Theorem 7.6 in |PRt §7.2]), Gf is equal to the group generated by its unipotent 
elements. (In the archimedean case, Gf is a connected Lie group.) Let G be a finite 
topological central extension of Gf- (For the definition of a topological central extension, 
the reader is referred to [Mo2, Chapter 1].) 

Take a maximally split Cartan subgroup H of G (which is also defined over F), and 
let A be the maximal split torus inside H. Let A be the inverse image of Af in G. 

We denote the Lie algebras of the groups G, H, A, G, A by g, fj, a, gp, ftp respectively. 
More precisely, g, f), a are F-Lie algebras and gp and dp are F-forms of the corresponding 
algebras. 

Let A be an absolute root system associated to f), and let £ be the restricted (or rel- 
ative, as some people call it) root system associated to a. The root space corresponding 
to any a G A is denoted by g Q , and the coroot for a is denoted by H a . 

We fix a G-invariant symmetric bilinear form (•, •) on g, normalized in such a way that 
if for any two roots (3, 7 we define (/3, 7) to be (Hp, H^), then ((3, (3) = 2 for any long root 
(3. All roots of simply-laced root systems are considered as long ones. For f)p = f) H gp 
we have 

1}f = % © t F 

where tp is the orthogonal complement of dp in f)p with respect to the invariant symmet- 
ric bilinear form (•, •) introduced above. 

As usual, A + and S + denote the set of positive roots. We assume A + , when restricted 
to a, contains S + . Let A# and E# denote the bases for positive roots in the correspond- 
ing root systems such that A#, when restricted to a, contains 

The choice of a positive system determines a fixed Borel subalgebra b of g (or a minimal 
parabolic p m i n of gp). A parabolic subalgebra of g (respectively, of gp) is called standard 
if it contains b (respectively, p m i n ). We identify the standard parabolic subgroups and 
subalgebras of g (respectively of gp) by subsets of A B (respectively of S B ) they naturally 
correspond to. More precisely, for any subset $ C A#, by p$ we mean the standard 
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parabolic subalgebra of g which contains all root spaces g a for any a e A + as well as any 
0_ a ' such that a' G A + is in the semigroup generated by $. The definition will be similar 
for the parabolic subalgebra (this time of jjf) corresponding to any $CE g . Later on, 
when there is no risk of confusion between the parabolics of g and g-p, we use the more 
concise notation p$ for both situations. Also, again for simplicity and when there is no 
risk of ambiguity, Lie subalgebras of both g and gw may be denoted by Gothic letters in 
a similar way; i.e. without a subscript "F" in the case of a Lie subalgebra of gw- 

The standard parabolic subgroups of G and G are denoted by P$ and P$ respectively. 
Note that a parabolic subgroup of G means the inverse image of a parabolic subgroup of 
Gf in G. In other words, if the short exact sequence 

(1-1) 1 ► F — G — G ► 1 

(where F is a finite subgroup of the center of G) represents our topological central 
extension, then a parabolic P of G is of the form P = p~ l (P^) for an F-parabolic P of 
G. It follows from [Du, II 11 Lemme] (or from an explicit construction of the universal 
central extension by Deodhar |De| Section 1.9]) that topological central extensions of 
semisimple groups over local fields are split over the unipotent subgroups 1 and therefore 
one can have an analogous "Levi" decomposition in G as well; i.e. if P = p _1 (Pjf) and 
P = LN is a Levi decomposition of P (with L being the reductive factor and N being 
the unipotent radical) then one can express P as P = LN, where: 

1. L = J 9- 1 (L F ). 

2. iV C j9 _1 (Nf) is a normal subgroup of P and the map 

p : iV N F 

is an isomorphism of topological groups. (Existence of N is what follows from 
the results of Duflo or Deodhar cited above.) 

We may as well drop the subscript $ of the standard parabolics Pq, and P$ when there 
is no risk of confusion about the identifying subset <3> of simple positive roots. Moreover, 
whenever a parabolic subgroup or subalgebra is standard, we assume that the Levi de- 
composition considered above is also standard. 

Let K be an arbitrary locally compact group. Then the center of K will be denoted 
by Z{K). Unitary representations of K are defined in the usual way (see jMaclj ) and 
denoted by Greek letters. The unitary dual of K (which is the set of equivalence classes of 
the irreducible unitary representations of K endowed with the Fell topology) is denoted 
by K. Now suppose K' is a closed subgroup of K and let a, a' be unitary representations 
of K, K' respectively. Then, as usual, Res^/O" and Ind^cr' mean restriction and unitary 
induction. If there is no ambiguity about K, we use the simpler notation <j\k* instead of 
Res^/cr. 

Suppose [ is an arbitrary Lie algebra. Then the center of \ is denoted by 3(C). 



In fact an abstract (and hence topological) universal central extension of Gf splits over the maximal 
unipotent subgroup. I am indebted to Gopal Prasad for pointing to reference IDe) . 
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2. Heisenberg group and Weil representation 

2.1. Heisenberg groups and algebras. Let n be a positive integer. An F- Heisenberg 
group (or simply a Heisenberg group) H n is a two-step nilpotent 2n + 1-dimensional 
F— group which is isomorphic to the subgroup of GL n+2 (F) of unipotent, upper-triangular 
matrices which do not have nonzero elements outside the diagonal, the first row and the 
last column. This group has a one-dimensional center. We denote the Lie algebra of H n 
by f) n . Since H n C GL n+2 (F), f) n sits in the usual way inside 0t n+ 2(F). 

The Lie algebra f) n of H n can also be described more abstractly as 

f>n = W n © 3 

with the Lie bracket introduced below, where W n is a 2ra-dimensional space endowed with 
a nondegenerate symplectic form < -, ■ >, and 3 is the center of f) n , a one-dimensional Lie 
subalgebra. We fix a Lie algebra isomorphism between 3 and F, and denote the element 
in 3 corresponding to 1 G F by Z. The Lie bracket on W n is defined as follows: 

for every X, Y e W n , [X, Y] =<X, Y>Z. 

Viewing fj n as a subalgebra of g[ ri+2 (F), the exponential map of GL n+2 {¥) gives a bijec- 
tion between f) n and H n , and Z(H n ) is equal to exp(3). This bijection is the exponential 
map of H n . 

2.2. Stone-von Neumann theorem. We briefly mention the structure of irreducible, 
infinite-dimensional unitary representations of a Heisenberg group. For a more detailed 
discussion, the reader can study |Hw2j . |CG| or |Tay[ Chapter 1] for instance. 

The Stone-von Neumann theorem states that for any nontrivial unitary character of 
Z(H n ), up to unitary equivalence there is a unique infinite-dimensional irreducible uni- 
tary representation of H n having this central character. We will describe an explicit 
realization (the so-called Schrodinger model) for this representation below. 

Consider an arbitrary polarization of W n , i.e. a decomposition of W n as a direct sum 

(2.1) W n = y„ © % 

of maximal isotropic subspaces. It is possible to choose bases {X\, . . . , X n } and {Yi, . . . , Y n } 
of y n and t) n respectively such that 

(2.2) <X i ,Y j >=S iJ . 
There is an isomorphim 

(2.3) e : ?n « t)* n 

obtained via the symplectic form. Z(H n ) m exp(F) and any nontrivial multiplicative 
(unitary) character of Z(H n ) is equal to x ex P~\ f° r some x> where x can be any 
nontrivial additive (unitary) character of 3(f) n ) ~ F. Such characters x are i n one-to-one 
correspondence with elements of F — {0}. For example, if F = R, x will be of the form 
x{x) = e lxy for some j/Gl- {0} (here i = \J— 1). This follows from the well-known fact 
that the unitary dual of a local field is identifiable to itself (see |Welj ). 
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The (irreducible) representation p of H n with central character x may be realized on 
the Hilbert space 

n p = L\t) n ) 

as follows. Let y = Yli=i Vi^i e Then H n acts on H p as 

(p(e tXi )f)(y) = x(-t yi )f(y) 

(2.4) (p(e tYi )f)(y) = f(y + tYi) 

(P(e tz )f)(y) = X (t)f(y). 

This representation is irreducible and unitary with respect to the usual inner product of 
Hp. 

Notation. Henceforth, whenever a locally compact group N is isomorphic to a Heisen- 
berg group, we denote the subset of N consisting of the family of the infinite-dimenional 
representations constructed above by N Q . 

2.3. The Weil representation. Let Sp(W n ) be the symplectic group associated to W n . 
When F = C, take Mp(W n ) = Sp(W n ), and otherwise let Mp(W n ) be the (metaplectic) 
double covering of Sp(W n ) (see |We2j ). Mp{W n ) acts through Sp(W n ) on H n w exp(f) n ) 
(or equivalently on f) n ) as follows. For any w®z6 W n © 3, 

One can consider the semidirect product Mp{W n ) x H n . We have (see |We2j ) : 

Proposition 2.3.1. Let x be any nontrivial additive character of ¥. Then the irreducible 
unitary representation p of H n with central character x can be extended to a unitary 
representation (still denoted by p) of Mp(W n ) x H n ; i.e. for any g £ Mp(W n ) and any 
h £ H n : 

p{g)p{h)p{g~ l ) = p{ghg~ l ). 

3. Construction of H-tower groups 

3.1. The Heisenberg parabolic. We use the notation of section fOl Let f3 be the 

highest root in A with respect to the positive system chosen in section H~2*l and let Hz be 

the coroot for (3. Let (•, •) denote the G-invariant symmetric bilinear form introduced in 
that section; i.e. (/3, (3) = 2. One obtains a grading 

= 00i 

where Qj is the j-eigenspace of ad(iLj); i.e. 

Sj = {Xeg:[H $1 X\=jX}. 

Note that in fact Qk = {0} for \k\ > 2. The Jacobi identity implies [fji,0j] C Qi+j. Note 
that Qi = 3_i = {0} if and only if g = sl 2 . 

In the next proposition and at some points of this paper, we will consider g as above 
such that: 

(3.1) o q 7^ s[ 2 and $ is defined over F. 
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Condition (J3.1)) is not a crucial assumption. In fact the author believes that it can be 
removed. See the discussion in the remark after Proposition 13.1.11 

Proposition 3.1.1. Let g satisfy condition V3.1)) . Then 

1- 0o © 01 © 02 i> s a parabolic subalgebra, Qi © g 2 is its nilradical, and 

[[flo, So], 02] = {0}. 

The nilradical is a Heisenberg Lie algebra with center Q2 and symplectic space 
0i. One can describe the symplectic form <•, ->x on gi using the Lie bracket as 
follows: fix X2 G Q2 — {0} and let <•, ->i be such that 

for every X, Y G 0i , [X, Y\ =<X, Y> 1 -X 2 . 

Then <■, ->i will be nondegenerate, and the (adjoint) representation o/[go,0o] on 
0! will be a symplectic representation with respect to <-, ->x- 

2. When g is not of type A; (I > 1 ), the parabolic of part 1 is characterized by 
the subset Ab — {/3} of Ab for the unique simple root [3 G A B which satisfies 
{(3,(3) 7^ 0. For type Ai (I > 1), there are two such simple roots f3',/3", and the 
parabolic corresponds to A B — {/?', (3"}. 

3. When g is not of type A; (I > 1), (3 corresponds to a simple restricted root (still 
denoted by f3). For g of type Ai (I > 2) and gp non-split, the pair {(3 1 ', [3"} corre- 
sponds to a simple restricted root (3 in the restricted root system. 

Let the sets S C A# and T C S# be defined as follows. Set S = {(3} when g is not of 
type Ai (I > 1) and S = {f3',/3"} otherwise. Also, set T = when gp is split and 

g is of type Ai (I > 1), and T = {(3} otherwise. 

4. The parabolic Pa b -s is defined over ¥, and (Pa s -s , )f = Ps B -r- Its nilradical is 
a Heisenberg algebra. Pa b -s is a maximal parabolic when g is not of type Ai, 
(1>1). 

Proof. See jHWl §2] and jE3 §10]. □ 

Remark. The assumption that g satisfies condition (J3.1)) holds for all but a very small 
class of Lie algebras g. Fox example, when F = R, the cases where (3 is not defined over R 
are f4(_2o), e 6(-26), 5 P(p, <?), an d sl(n, H). The notion of rank in this paper is based on the 
existence of a unipotent subgroup of G which is expressible as a tower of extensions by 
Heisenberg groups (see Definition I3.2.5|) . and therefore it is not applicable to the groups 
mentioned above. However, in all of these groups there does exist a similar structure 
which is called an OKP subgroup in |HRWj . Therefore in principle the main results of 
this paper (especially Theorem 15. 3. 2|) should generalize to groups associated with these 
real forms. However, addressing the technical problems which arise with including those 
cases in this paper makes it much more technical. To keep our presentation as simple 
and uniform as possible, we do not include those special cases in this paper. The author 
intends to deal with those cases elsewhere. 
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Definition 3.1.2. The parabolic subalgebras Pa b ~s of g and ps s -T of gf or their corre- 
sponding parabolic subgroups Pa b -s ofG andP^ B -T = (Pa s -s)f °/Gf (or the parabolic 
of G which corresponds to P^b-t) ore called the Heisenberg parabolics. 

We tend to drop their identifying subscripts for simplicity when there is no risk of 
confusion. 

Table I below demonstrates the structure of gi as a representation of [go, go]- Here 
denotes the representation with highest weight w and W{ denotes the z-th fundamental 
weight of the corresponding Lie algebra. We use the notation of |Bou| Planche I] for 
numbering fundamental weights. See also jTI] for more explicit information. 






[0o, So] 


0i 


A, (1 > 3) 


Al_ 2 


© V* t 


B 2 


Ax 


V W1 


B 3 


Ai x Ai 




Bi (1 > 4) 


Ai x Bi_ 2 


V W1 ®V W1 


c 2 


Ax 




Q (1 > 3) 


C,-i 




D 4 


Ai x Ai x Ai 


v mi ®v mi ®v ai 


D 5 


Ai x A 3 




D, (1 > 6) 


Ai x Di_ 2 




E 6 


A 5 




E 7 


D 6 




E 8 


E 7 




F 4 


c 3 




G 2 


Ax 


v 3mi 



Table I 



3.2. The H-tower subgroup Nr of G. Let g be as in condition 1)3.1)1 . In this section 
we describe a nilpotent subalgebra of g (and another one of 0f) which is fundamental to 
our definition of rank. The construction of these nilpotent subalgebras is based on what 
is usually referred to as Kostant's Cascade. We show that this nilpotent Lie subalgebra 
is actually the nilradical of a parabolic subalgebra (see Propositions 13.2.31 and 13.2.4)1 . 
The unipotent subgroup of G which corresponds to this nilpotent subalgebra will play 
an important role in the rest of the paper. 

First assume that g splits over F; i.e. all roots of g are defined over F. Let p = [ © n 
be the Heisenberg parabolic of g, obtained by Proposition 13.1.1) with the usual Levi 
decomposition; i.e. [ is the Levi factor and n is the nilradical of p. In the case of 
orthogonal algebras the commutator [I, I] is not a simple Lie algebra. In fact when g is 
of types (I > 3) or (/ > 4), the commutator [[, f] is a direct sum of the form 

(3.2) [I,r]=st 2 ©s 

where s is simple. When g is of type D4, we have 



[[, [] =si 2 ©s[ 2 ©sl 2 . 
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Definition 3.2.1. Let m be defined as follows: 

I. If g is of types D/ (I > 4) or B; (/ > 3) then m is equal to the summand s given 
in ( El) . 

II. If g is of types A 2 , A 3 , B 2 = C 2 , B 3 , D 4 or G 2 t/ien m = {0}. 
III. Otherwise, m = [I, []. 

One can repeatedly apply Proposition 13.1.11 as follows. First we apply it to m. If m 
is nonzero, then m 7^ st 2 and Proposition 13.1 .11 guarantees the existence of a Heisenberg 
parabolic in m. Let p' be this parabolic of m, and let [' be the Levi factor of p'. Let 
m' be the subalgebra of [I', I'} which is defined in the same way that m was defined as 
a subalgebra of [I, I] in Definition 13.2.11 Then we apply Proposition 13.1.11 to m', and so 
on. This process can be repeated as long as Proposition 13.1.11 can be applied. As a 
result, we obtain a sequence Si, . . . , S r of subsets of A#, where each Si, defined as in 
part 2 of Proposition 13. 1. i} contains either a simple root or a pair of simple roots. Each 
Si corresponds to the "highest root" $ obtained in the i-th step. We also denote the 
sequence of Heisenberg parabolics by p 1 , . . . , p r , with the Levi decomposition 

(3.3) pJ = p©n j . 

Therefore, p 1 is the Heisenberg parabolic of g, p 2 is the Heisenberg parabolic of m, and 
so on. Each tv 7 is normalized by any P for j' > j and hence by n- 7 . For a similar reason, 
x\P acts trivially on the center of n J . Each x\P is isomorphic to a Heisenberg algebra f)^ 
for some <ij. Therefore the Lie algebra n 1 © ■ • • © n r is a tower of successive extensions 
by Heisenberg F-algebras. The following proposition is obvious. 

Proposition 3.2.2. n 1 © ■ ■ ■ © n r is equal to the nilradical of the parabolic subalgebra p r 
of q where 

r = A B - (5iU...U5 r ). 

Now assume g satisfies condition (J3.1|) but it is not necessarily F-split. Recall that the 
positive system for E is compatible with the one for A. We can apply Proposition 13. 1 . Il 
repeatedly again. Note that the number of possible iterations for a non-split F-form of 
g is often smaller than the number of possible iterations in the split case. (In certain 
groups with small rank, they may be equal.) This is because the successive Levi factors 
may fail to satisfy condition (|3.1|) . 

Part 3 of Proposition 13.1.11 yields a sequence T 1; . . . ,T S of subsets of S B (for some 
s < r) where each Tj contains a simple restricted root or a pair of them. Again we 
obtain a nested sequence 

p 1 D • • • D p s 

of Heisenberg parabolics (this time inside gv) and the nilradical of any p* is normalized 
by the nilradical of any p l when i < il . Therefore p 1 is the Heisenberg parabolic subal- 
gebra of Qf. We warn the reader that the new p l 's are different from those which appear 
immediately before Proposition 13.2.21 in fact the older p l 's are obtained from the newer 
p l, s by an extension of scalars. 
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- rl r,r 


(r 


>4) 


A--2,r-2 


2,(1) 
s±2r-l,r 


(r 


>3) 


2,(1) 
/1 2r-3,r-l 


2,(1) 
^2r,r 


(r 


>2) 


2,(1) 
/1 2r-2,r-l 


2,(1) 
^ 1 2r+l,r 


(r 


>2) 


2,(1) 
/1 2r-l,r-l 


-84,4 






C2,2 


-83,2 








B r ,r—1 


(r 


>5) 





r,r 

1 D (1) 
-^5,5 

-^4,2 
(i) 



2 D 



r+2,r 
(1) 



5,4 

in( 2 ) 

-^4,2 
1^(2) 



5,1 

2r>(2) 

-^5,2 

2n( 2 ) 

-^4,1 
3r)2 

1 77116 

^6,2 

2 7712 

^6,4 

-^7,7 
^4,4 



(r > 3) 



(r > 4) 



C) — l,r— 1 
^3,3 

1 D (1) 

2,(1) 
^3,2 



2,« 
-^3,1 



2,(1) 
^5,3 
(1) 



C 3 ,3 



6,6 



S 

LiJ 

r — 1 
r 
r 
2 
1 

L^J 

r — 1 
2 
1 

LiJ 
2 

1 
1 
2 
1 
1 
1 

3 
3 



_0F 

^3,3 

2 ,(1) 
^3,2 
2 ,(1) 
^2,1 
(1) 
3,1 



3,3 



-^4,3 
^2,2 



2 D 



(1) 
4,4 

1 Di 1 } 

-^5,3 
(1) 

4,3 

ir,(2) 

- Ly 2r,r 

in( J ) 

ly 2r+3,r 
2r>(2) 

2r)(2) 

6r)2 

1 po 

^,4 
^2,2 



(r > 4) 
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(r > 5) 
(r>3) 
(r>2) 
(r>3) 
(r>3) 



m F 



5 



r-2,r-2 



r-2,r-2 



- Ly r--l,r--2 

ir)(2) 

xy 2r^2,r-l 



1^(2) 



2r+l,r-l 
2r,(2) 

2jD (2) 



2r,r-l 



^5,5 



6,3 



-^7,7 



1 
1 
1 
1 
1 

LIJ 
1 

1 
1 

L^J 
1 

1 

L^J 

r- 1 
r 
r 
r 
1 
3 
3 
4 
1 



Table II 



9w 


TTlF 


s 


0F 


mi 


s 










1 


sl 4 (R) 








1 


s[„(R) 




(n > 5) 


8ln- 2 (R) 


L^J 


su(l,q) 


(9>1) 






1 


su(2,2) 








1 


su(r, g) 


(2 < r < q) 


5u(r — 1, g — 


1) 


r 


su(g,g) 




(3<5) 


su(q- l,q-l) 




so(l,g) 


(9>4) 






1 


so(2,g) 




(9>3) 




1 


so(3,g) 


(9>3) 






1 


so(4,4) 








1 


so(4,g) 


(9>5) 


so(2,q>- 2) 




2 


so(r, g) 




(5 < r < g) 


so(r -2,q-2) 


LiJ 


so(<7,g) 


(5<g) 


S0(q- 2,q- 


2) 


L^J 


sp 4 (R) 








1 


sp 2 „(R) 


(n > 3) 


sp 2 „- 2 (R) 




n - 1 


so*(6) 








1 


so* (8) 








1 


so*(2r) 




(r > 5) 


so*(2r - 4) 


L^J 


Oe,sp 4 ) 




s[ 6 (R) 




3 


0e,su 6 


x su 2 ) 




su(3,3) 


3 


(e 6 ,so(10) x u(l)) 




su(l,5) 




2 


(e7,su 8 ) 






so(6, 6) 


3 


(e 7 ,so(12) x su 2 ) 




so*(12) 




3 


(e7,e 6 x 


u(l)) 




50(2,10) 


2 


(e 8 ,so(12)) 




(e7,su 8 ) 




4 


(e 8 ,e 7 x 


su 2 ) 




(e 7 ,e 6 x u(l)) 


3 


(f 4 ,sp 3 x su 2 ) 




sp 6 (R) 




3 


(fl2,SU 2 X SU 2 ) 






1 













Table III 
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Tables II and III explain how Proposition 13 . 1 . ll is applied iteratively to gf For sim- 
plicity, the real and the p-adic cases are separated: Table II is for the p-adic case and 
Table III is for the real case. 

The column g F in Table II shows the Tits index of g F and the column trip shows the 
Tits index of trip. For any g F the number s is given too. Real exceptional Lie algebras 
in Table III are identified by the symmetric pairs ($3r, 6r). The only F-forms of g which 
appear in the columns g F of both of the tables are those for which g satisfies condition 
(|3.1|) . If an entry in the column corresponding to m F is equal to this means either 
that the Lie algebra m is equal to {0} or that the Lie algebra m is semisimple but it does 
not satisfy condition (|3.1|) . i.e. the highest root of m is not defined over F. Therefore one 
can use Tables II and III to see how many times Proposition 13.1.11 can be applied to a 
particular F-form of g. 

We denote the nilradical of the parabolic p- 7 of g F by f) J . It is an F-Heisenberg algebra. 

Proposition 3.2.3. The nilpotent Lie subalgebra f} 1 © • ■ ■ © f) s of Qv is equal to the 
nilradical of the parabolic subalgebra pr of g F where 

r = S B -(T 1 U...UT s ). 

Moreover, we have the following proposition, which describes the relationship between 
n*'s and ff' s. 

Proposition 3.2.4. Let pr be the parabolic subalgebra of Qf defined in Proposition ^. 2. 3i 
with Levi decomposition pr = tr © % ■ Then 

n r © F = n 1 © • • • © n s 

(where the n J 's are the same as those which appear in Proposition VS. 2. 2)) and thus rir ©F 
is equal to the nilradical of the parabolic pr' of g where 

r , = A B -(SiU...us' s ). 

Therefore Pp/ is an F-parabolic of G and (Pt')f — Pr ; where Pr is the parabolic of 
G which is associated to the set V C E# defined in Proposition 13.2.31 Pr will play a 
significant role in the rest of the paper. Its Levi decomposition can be written as 

(3.4) P r = L r Ar- 

Definition 3.2.5. Let U be the group ofF-points of a unipotent linear algebraic F-group 
U. U is said to be an u H-tower" group if and only if it satisfies one of the properties I 
or II below. 

I. U = {1}; i.e. U is trivial. 

II. U is isomorphic to a semidirect product U' k U" where 

i. U" is an F-Heisenberg group and is the group of F-points of an algebraic 
F-subgroup U" o/U. 

ii. U' is the group of F -rational points of an algebraic subgroup U' o/U and the 
action ofU' on U" in the semidirect product U' k U" comes from an algebraic 
action (defined overF) o/U' on U" by group automorphisms o/U" (which, 
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a fortiori, leave elements of Z(XJ") invariant and act on the symplectic space 
XJ" / Z(XJ") via symplectic operators) 2 . 
iii. U' is an H-tower group. 

Therefore an H-tower group U can be expressed as a tower of successive extensions: 

(3.5) U = H l ■ H 2 ■ ■ ■ H l = H 1 x (• • • x (H*' 1 x H l ) ■ ■ ■ ) 

where each W is a Heisenberg group; i.e. W = for some dj (see section EHJ). 

Remarks. 

1. The name "H-tower" is chosen so that it reminds the reader that the group is a tower 
of extensions by Heisenberg groups. 

2. Real H-tower groups form a subclass of OKP groups defined in jHRWj. 

3. The number t in ()3.5|) is referred to as the height of the tower of extensions. It will 
be denoted by ht(U). 

4. Consider an H-tower group U which is expressed as in (J3.5|) . for any j G {1, ht(U)} 
we denote the quotient group _£P ■ ■ ■ H ht( - U ^ « U /H 1 ■ ■ ■ W~ x by Uj. Uj is also an H-tower 
group. 

If in (J3.5|) we take U = N-p and = exp fj J ', then by Proposition 13.2.31 we have 
Proposition 3.2.6. N-p is an H-tower group andht(Np) = s, where s is as in Proposition 

ma 

4. Rankable representations of H-tower groups 

4.1. Oscillator extension and rankable representations. Fix an H-tower group U, 
expressed as a tower of successive extensions as in (J3.5|) . Let \i De a nontrivial additive 
character of F, and consider the unitary representation pi of H 1 with central character 
Xi (see section |2~2J) . When restricted to the inverse image (in the metaplectic group) of 
the maximal unipotent subgroup of the symplectic group, the Weil representation factors 
through a representation of the maximal unipotent subgroup of the symplectic group. 
(In fact the two-fold central extension of the symplectic group splits over the maximal 
unipotent subgroup and, more importantly, this splitting is unique. This holds because 
any two splittings would differ by a finite-order character, whereas the maximal unipotent 
subgroup is a divisible group and therefore it does not have such characters.) Therefore, 
any representation p of any Heisenberg group H n with central character x ( see section 
12. 2 j) is extendable to the unipotent radical of any Borel subgroup of the symplectic group 
Sp(W n ). This implies that the representation p\ of H 1 is extendable (in at least one 
way) to U. We still denote this extension by p x . Note that the extension of p\ is not 
necessarily unique, as one can for example twist it by a one dimensional representation 
of U which is trivial on H 1 . However, since the extension of p to the metaplectic group 
is unique, we can distinguish the extension of p± obtained by the restriction of the Weil 

2 This means that there is an F-homomorphism of algebraic groups $ : U' Sp(U"/Z(U")) and 
consequently $([/') C Sp [U"/Z{U")) . 
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representation without ambiguity. Let p\ mean this specific extension. 

In a similar fashion, for any j > 1 one can construct a representation of 

Uj = U/H 1 ■ ■ ■ W- 1 &H j --- H ht{u) 

as follows. We take an arbitrary nontrivial additive character Xj of F, and the represen- 
tation pj of Hi with central character \y As before, we use the Weil representation to 
extend pj to Uj. Since 

Uj « U/H l ---H j -\ 

this representation can be extended to U, trivially on H 1 ■ ■ ■ iiP -1 . Thus using pj we have 
constructed a representation of U. We still keep the notation pj for this representation. 

Definition 4.1.1. Let U be an H-tower group described as in \3.5)) . A representation of 
U is called "rankable" if and only if it is unitarily equivalent to a tensor product of the 
form 

Px <S> P2 <8> • ■ ■ <8> Pk 

for some k < ht(U), where pj is the representation ofU which is obtained (using the Weil 
representation) by extending the irreducible representation of with central character 
Xj- The rank of a k-fold tensor product (including the case k = 0, i.e. the trivial 
representation) is defined to be k. If a is a rankable representation of rank k, we write 
rank(a) = k 

Remark. It follows from Corollary 14.2.31 that rank(cx) is well-defined; i.e. it is an 
invariant of the unitary equivalence class of a. 

4.2. Kirillov theory for rankable representations. We denote the coadjoint orbit 
attached to an irreducible unitary representation a of a nilpotent group by O*. (See 
|Krlj . |Kr2l Chapter 3], |CGj or |Kit §2.2] for elaborated treatments of Kirillov's orbital 
theory.) In his seminal paper |Krlj . Kirillov developed his method of orbits for simply 
connected nilpotent real Lie groups. However, later in his 1966 ICM lecture he explained 
that essentially the same theory can be applied to algebraic unipotent groups over p-adic 
fields. See also |Mol| §4] for more details. 

Lemma 4.2.1. Let N be the group ofF-rational points of a unipotent linear algebraic F- 
group N. Let a be an irreducible unitary representation of N and let O* be the coadjoint 
orbit attached to it. Then O* is an analytic manifold in the sense of jSil Chapter III, 
§2]. 

Proof. The lemma follows immediately from |PRt §3.1, Corollary 2]. □ 

Lemma 14.2.11 implies that one can speak of the dimension of a coadjoint orbit. (The 
transitivity under the group action implies that the dimension is the same around every 
point.) For any coadjoint orbit O*, let dim O* denote its dimension. 

Lemma 4.2.2. Let N be a unipotent algebraic group such that N = N' x N" as algebraic 
¥ -groups ( and the action ofW on N" is defined over F ) and let N = N' x iV" be the group 
of ¥ -rational points o/N (where N', N" are the F-points o/N', N"^. Let the Lie algebras 
of N',N" be n',n" respectively. The Lie algebra of the semidirect product N = N' k N" 
is n = n' © n" as a vector space, and we have a canonical isomorphism of dual spaces 
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n* = n'* © n"*. Let a', a" be irreducible unitary representations of N',N" respectively, 
such that 

1) a' is an irreducible unitary representation of N' , extended trivially on N" to a 
representation a' of N . 

2) o" is an irreducible unitary representation of N" which extends to a unitary rep- 
resentation a" of N. (In other words, Res^cr" = a".) 

Then a' © a" is an irreducible representation of N and 

dim Ol, m „ = dim O*, + dim <D*„. 

Proof. Consider the map j defined as the composition 

N ^ N xN ^ N/N" xN ^N' xN 

where the leftmost map is the diagonal embedding and the middle map (i.e. the map 
iV x iV i— > N/N" x N) is projection onto the first factor (i.e. it is given by (ni,ri2) i— > 
(niiV", 712)). Observe that 

(4.1) or' © or" = Res^ V©a". 

It is easily seen by Mackey theory |Macll Theorem 3.12] that a 1 © a" is irreducible. 
Now take a maximal chain of analytic subgroups 

N' x N = N° D N 1 D N 2 D ■■■ D j(JV) 

such that each N^ has codimension 1 in iV 3 ' -1 . It follows from (|4.1|) that if a' © 5"" is 
considered as a representation of iV J , then 

Res^ a © a 

is an irreducible representation of iV- 2 . Now it follows from the statement of Theorem 
2.5.1 of |CG| that for any such j we are in the situation of part (b) of that theorem. But 
then Proposition 1.3.4 of |CG| implies that in this case, the dimension of the coadjoint 
orbit of a' © or" considered as a representation of N^ is the same as the dimension of the 
coadjoint orbit of a' © a" considered as a representation of N^~ x . Consequently, 

dim 0* /(| -„ = dim 0%, mll . 

Now consider the map q such that 

q : N' x N" ^ N' x N 

which is defined to be the identity map N' > N' in the first component and the injection 
N" C in the second component. Then 

R^xiv"/^" = a '® a " 

which is an irreducible representation. Again a similar argument (taking a maximal 
chain, etc.) proves that 

dim 0*,a-„ = dim „. 
But the coadjoint orbit 0* a ,^ a „ equals O*, x O*,,, which completes the proof. □ 

One can apply Lemma r4.2.2l to any H-tower group iteratively and obtain the following 
result. 
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Corollary 4.2.3. Let U be an H-tower group as in (t.9.,5)) . Let 

o = pi <g> p 2 ® ■ ■ ■ ® Pk 

be a rankable representation of U. (Recall that pj is extended from a representation of 
IP with central character Xj ■ ) Then a is irreducible. Moreover, as an analytic manifold, 
the coadjoint orbit O* has dimension 2(n\ + • • • + Uk), where 

dimf}- 7 — 1 



5. Main theorems 

5.1. Technical issues of central extensions. Let G, G,$j be as in section ll~2l such 
that g satisfies condition (J3.1j) . Let P be the Heisenberg parabolic of G (see Definition 
13 . 1 . 2j) . Suppose P = LN is the Levi decomposition of P. Therefore N is a Heisenberg 
group. As in section l2~2l let p be the irreducible unitary representation of N with an ar- 
bitrary nontrivial central character \. Let Nr be the H-tower subgroup of G constructed 
in section EIH and assume ht(iVr) > 1. 

This section is mainly devoted to a technical issue which arises with central extensions 
of G. The main result is Proposition 15.1.11 To avoid being distracted from the main 
point of the paper, the reader may assume Proposition 15.1.11 and go on to section 15.21 

Proposition 13.1.11 implies that the adjoint action provides a group homomorphism 
[L, L] i— > Sp(N / Z(N)). Our next aim is to somehow apply Proposition 12 .3 . ll and extend 
p to a representation of a larger group which at least contains N-p. One natural candidate 
could be [P, P] . However, when F ^ C, in order to extend the representation p of iV to 
[P,P] we need to know that indeed the group homomorphism [L, L] i— > Sp(N/Z(N)) is 
a composition of a group homomorphism into the metaplectic group, i.e. 

[L,L]» Mp(N/Z(N)), 

and the projection map 

Mp(N/Z(N)) h-> Sp(N/Z(N)). 

This issue was also dealt with in |Torj (see |GSt Section 3.3]). We will explain it more 
clearly below. In fact it can be seen that in some cases, if Pp = LpNp is the Heisenberg 
parabolic of Gp, then [Lp, Lp] does not act as a subgroup of the metaplectic group, and 
therefore extending p to [Pp, Pf] may not be possible. An obvious example is when Gp 
is the symplectic group. A less obvious example is the split group of type F 4 . (Actually, 
in both cases the extension is possible if we use the metaplectic covers. This may be 
seen after some simple calculations. For real groups one can use the results of |Adj which 
tell us when a covering of a real simple Lie group splits over embedded root subgroups 
SL2(M). However, here we do not rely on such calculations.) 

To overcome this difficulty, we will see below that we have to consider an appropriate 
finite (topological) central extension of G instead of G itself. Note that a representa- 
tion of G can trivially be considered as a representation of its central extension, and we 
can always study this extension instead of the original G because the central extension 
has an H-tower subgroup too, which is identical to and is essentially obtained by 
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exponentiating the corresponding nilpotent subalgebra tir of Qw- This follows from what 
was explained in section 11.21 as well, that a universal topological central extension of Gf 
splits over the maximal unipotent subgroup (see |Du[ II 11 Lemme] or |De| Section 1.9]). 
(In fact the splitting is unique because any two splittings would differ by a finite-order 
character, whereas any unipotent subgroup is a divisible group and therefore it does not 
have such characters.) Therefore, as far as it concerns the main results of this work, we 
can substitute G with such a "good" central extension of G, without any loss of generality. 

To show the existence of this "good" central extension for an arbitrary G, first we 
show its existence for the group G%. That is to say, we show that we can find a finite 
topological central extension Gf of Gf in which the representation p of N can be ex- 
tended to a larger subgroup. We warn the reader that we may (and will) think of N as 
a subgroup of both Gf and Gf at the same time, since the latter group has a subgroup 
identical to the subgroup N of the former one. 

Note that once we find Gf, a simple argument implies that the required extension 
exists for G as well. In fact the existence of a universal topological central extension of 
Gf implies that there will be a finite topological central extension which covers both G 
and Gf- This extension will be the one in which the extension of p to a larger subgroup 
is possible. 

Next we show that we can find Gf- Let 

1 ► F — > G F — G F ► 1 

be a finite topological central extension of Gf- Let P = LN be the Heisenberg parabolic 
of G with its usual Levi decomposition. Let L = ]3 _1 (Lf) and let iV be the unipotent 
radical of P = p _1 (Pf). The group [L,L], the derived subgroup of L, is a product of 
isotropic and anisotropic factors (see |Tort §8.19]). Let L 1S be the isotropic factor of 
[L, L]. Let M be the simply connected subgroup of [L, L] whose Lie algebra is equal 
to m, where m is the Lie algebra introduced in Definition 13.2.11 (The simply connect- 
edness of M follows from [Tor, §8.19].) It follows from ht(iV r ) > 1 that M C L is . Let 
M = p- 1 (M ¥ ). 

Recall that the adjoint action provides a map 

M (-> Sp(N/Z(N)). 

(See Proposition 13 . 1 . fl ) Let us call the inverse image of any subgroup M' of M inside 
the metaplectic group Mp(N/Z(N)) the metaplectic extension of M' . It follows from 
|Tort §8.25 and §9.3] that if the F-points of the minimal nilpotent G-orbit of g contain 
a GF-admissible orbit (see |Torl §3.17] for the definition of an admissible orbit), then 
there exists a closed subgroup M' of M, which is a normal subgroup of L, such that 
p(M') D (where is the subgroup of Mf generated by its F-rational unipotent 
elements) and the metaplectic extension of M' is trivial. (Note that in fact p(M') D Mf, 
because by |Tor| §8.19], M is simply connected and therefore M| = Mf (see [PR] §7.2]).) 
This means that one can extend p to M' ■ N . We keep the notation p for such an ex- 
tension. We will see below that the extension of p is actually uniquely determined on a 
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slightly smaller group, and therefore using p to denote the extension is not ambiguous. 

Tables in the end of jTorj determine which groups Gf have finite topological central 
extensions Gf with admissible orbits. Among all F-forms of groups G whose Lie algebras 
q satisfy condition (j3.1|) . the only groups which do not have coverings with admissible 
orbits are groups of Tits index B r r and B r r _ x in the nonarchimedean case and SU(p, q) 
and SO(p,q) when p + q is odd in the archimedean case. In these cases, however, the 
metaplectic extension of Mp is trivial by |Li2t Lemma 2.2] and [Ka2 Lemma 7]. Therefore 
if M' = Mp then p extends to M'. This settles the issue of existence of an appropriate 
covering for Gf- 

For a general G we can take G as a finite topological central extension which cov- 
ers both G and Gf. Let M be the inverse image of M' in G. Obviously extending 
p to M ■ N is possible. M has a closed finite-index subgroup Mi which is perfect 
(i.e. [Mi, Mi] = Mi). Mi is found as follows. Define the sequence M<°) = M and 
M (i) _ [AfC*- 1 ), Af( i-1 >]. Then each M^ is a subgroup of M' of finite index and the 
indices are uniformly bounded (in fact one upper bound is the size of the kernel of the 
map G i — > Gf) . Therefore there is some i such that M (io) = M (io+1) ; i.e. M (io) is perfect. 
Let Mi = M(*°), the closure of M^ inside M'. Then Mi is perfect by fTcr! §8.11]. 

The extension of p to Mi ■ N is unique. In fact the projective representation of Mi is 
uniquely determined by the relation 

p(m)p(n)p(m~ 1 ) = p(mnm~ ) for any n G N,m G Mi 

and since Mi is perfect, this projective representation lifts to a linear representation of 
Mi in a unique way. 

The discussion in the previous paragraph proves the following result. 

Proposition 5.1.1. Let G be as in section WM such that q satisfies condition hS. Let 
G be a finite topological central extension of Gp represented as in M.l)) . Let P, L, N, p be 
as in the beginning of section \5.1\ Then there exists a finite topological central extension 
G of Gp which covers G as in 

1 > F — i-» G — G ► 1 

and a (closed) subgroup P\ = Mi ■ N of G such that 

i. p o p(M l ) = M F and p o p(N) = N F . 

ii. N is the unipotent radical of p~ l {P); i.e. p : N i— > iV is an isomorphism. 

iii. The representation p (which can naturally be thought of as a representation of N 
too) can be extended in a unique way to a representation of P\. 

iv. Mi is perfect; i.e. [Mi,M x ] = M x . 

v. Pi is a normal subgroup ofp~ 1 (P). 
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5.2. Mackey analysis. In this section we assume that the notation is the same as in 
section |5~T1 We also assume (without loss of generality) that 

(5.1) o G is equal to the extension G of Proposition 15. 1 . 11 and [G, G] is dense in G. 

Note that if [G, G] is dense in G, then from |Tor[ §8.11] it follows that in fact [G, G] = G. 
The assumption [G, G] = G is not a crucial one. This is because one can always find a 
closed finite-index subgroup G\ of G such that G\ is perfect and G = G\- 2(G), and once 
a representation n of G is understood on G\, It is easy to describe it as a representation 
of G. 

Assume condition ()5.1|) holds. Let P± C G be the subgroup given in the statement of 
Proposition 15.1.11 Then we can write Pi = Mi ■ N, where Mi is defined in Proposition 
15.1.11 and N ^ N is the unipotent radical of the Heisenberg parabolic P of G which 
appears in parts ii and iii of Proposition 15.1.11 

Let 7r be a unitary representation of G, without nonzero G-invariant vectors. Consider 
the restriction of 7r to P. Recall that Z(N) means the center of N. By Howe-Moore 
theorem |HMj . 7r does not have a nonzero iJ(iV)-invariant vector either. Therefore in 
the direct integral decomposition of 7r as a representation of N, the spectral measure is 
supported on N (see the notation introduced at the end of section |2~2|) . 

Elements of Pi commute with elements of Z(N). Therefore under the coadjoint action 
of P on the unitary characters of Z(N), Pi lies within stabp(x), the stabilizer (inside P) 
of any nontrivial additive unitary character x of Z(N). In fact all these stabilizers are 
identical groups. We denote this common stabilizer group by J. 

The action of P on Z(N) (and also on its unitary characters) has only a finite number of 
orbits. Consequently, since n has no nonzero G-invariant vectors, by elementary Mackey 
theory (see Theorems 3.11 and 3.12 of |Maclj ) the restriction of 7r to P can be expressed 
as a finite direct sum 

tt|p = (J)lndj(7j 

i 

where each <7j is a representation of J which, when restricted to N, is a direct integral of 
representations isomorphic to the representation p^ of N (defined in sect ion |2~2*|) with some 
central character \i- However, each pi extends in exactly one way to a representation 
of Pi. We still denote this extension by pi. Again by Mackey theory we can write the 
restriction of <7j to Pi as 

CilPi =Vi® Pi 

where is indeed a representation of Mi extended trivially on iV to P\. Therefore we 
have proved the following result. 

Lemma 5.2.1. Let G, P, L and N be as in the beginning of section \5.1\ so that G satisfies 
condition 115.1)) . Let it be a unitary representation of G without a nonzero G-invariant 
vector. Then tt can be written as a finite direct sum 



(5.2) 



7T|p = (J)lndj(7i 

% 
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where each o~i is a representation of J such that 

(5.3) cr l | Pl =Vi® pi 

and Vi factors to a representation of M\. (In other words, Pi is trivial on N .) 

Recall the construction of the H-tower subgroup U = (see (|3.4)l ). In the notation 
of ()3.5|) we have H 1 = N. The quotient group U2 = N-p/N is identical to the H-tower 
subgroup which is constructed for M. By Proposition 15.1.11 the latter H-tower group 
should lie within Mi. 

Lemma 5.2.2. Let p be the extension to Pi of the irreducible representation of N (which 
is also denoted by p) with an arbitrary nontrivial central character \ (see section HO)) . 
Let N r be as in \3.J$ - Then the restriction of the representation p to N r is supported on 
rankable representations of Nr of rank one. 

Proof. The uniqueness of extension of p to Pi implies that on Mi, this extension should 
be identical to the extension obtained by restriction of extension of p to the metaplectic 
group acting on N/Z(N). Therefore the lemma follows from Definition 14.1.11 

□ 

Notation. Let K be an arbitrary (abstract) group and let K' be a subgroup of K. Let 
a be a representation of K' on a vector space 7i. Let a G K. Then by o~ a we mean a 
representation of the subgroup K' a = aK'a~ x on TC defined as follows: 

(5.4) for every x G K' a , a a (x) = a{a" l xa). 
In particular, if K' = K, then a a is a representation of K. 

At this point we recall Mackey's subgroup theorem. We will use it in sections 15.41 
and 15.51 Giving a precise statement would require some definitions and would distract 
us from the main point of this paper. Therefore, for a detailed discussion we refer the 
reader to |MacH Theorem 3.5] or [Mac2, Theorem 12.1]. 

Theorem 5.2.3. (Mackey's Subgroup Theorem) Let K be a locally compact group and 
let K', K" be its closed subgroups. Assume that K, K' , K" are "nice" (see the above refer- 
ences for details). Let o be an irreducible representation of K' . Then the representation 

Res^//Ind^/cr 

has a direct integral decomposition supported on representations t v ,v G K, where 

t v = Ind^// ni ^Res i ^y, rLK v<7 1 '. 

Here 

K' v = vK'v' 1 

and a v is defined as in \5.4\j - 

The following elementary lemma, which essentially follows from Mackey theory too, 
will help us in sections 15.41 and 15.51 

Lemma 5.2.4. Let K' C K be two arbitrary locally compact groups such that K' is a 
closed normal subgroup of K. Let a be a unitary representation of K which acts on the 
Hilbert space TC. Suppose 

Res^-,(T = ai®a 2 
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where o\ and 02 are unitary representations of K' such that for any a G K , we have 

Hom^^Vs) = {0}. 

(Here cr" is defined as in \5.J$ and Hornier", a 2) means the space of K' -intertwining 
operators from of to o 2 .) For i G {1,2}, let Hi be the closed a (K 1 ) -invariant subspace 
of H which corresponds to the summand o~i . Then each Hi is actually invariant under 
o(K). 

Proof. Let a G K. Then the linear operator T defined as 

T:Hi i-> H 
T(w) = a(a)w for all w G Hi 

is an element of Hom^/(o"", a). The orthogonal projection V of H onto H2 is an element 
of Hom^-/(cr, (T2). By the hypothesis of the lemma, one should have ?oT = 0. Therefore 
a(a)Hi C Hi. 

□ 

5.3. Statement of the main theorems. We state the main theorems in this section 
and prove them in sections 15.41 and 15.51 

Throughout this section, we assume G, Gf and q are as in section 11.21 and g satisfies 
condition (|3.1|) . Moreover, we assume N r is the H-tower subgroup of G (see equality 
(|3.4j) ). described as in (|3.5|) . 



Theorem 5.3.1. Let n be a unitary representation of G. Then in the direct integral 
decomposition 

7T|jVr = / Tdn(r) 

the support of the spectral measure \x is inside the subset of rankable representations of 
N r . 

In the next theorem the subset of rankable representations of rank k of will be 
denoted by N r (k). 

Theorem 5.3.2. Let it be a unitary representation of G on a Hilbert space H^. Consider 
the direct integral decomposition 

■K\ Nv = / Tdfi(r) 

and let be the projective measure corresponding to this decomposition. Let Nr(Tr) be 
the support of it in this direct integral decomposition. Set 

Hi = V^Nr(j) n iVr(Tr)) • 7U for any 0<j< ht(JV r ). 
Then for every j such that 

jG{0,l,2,...,ht(iV r )}, 
Hi is a G -invariant subspace ofH n . The direct sum of all these subspaces is equal to 



A NOTION OF RANK FOR UNITARY REPRESENTATIONS 



21 



Remark. It is clear that for V^Npij) H Np(n)) to make sense we need to show that 
the sets Np(j) H Np(ir) are indeed Borel subsets of Np. In fact one can show that the 
set of rankable representations of a given rank of an H-tower group U = H 1 ■ ■ ■ H l can 
be constructed with a finite number of set-theoretic operations on open and closed sets 
of the unitary dual of U. Here is a sketch of the proof. Any rankable representation a is 
a tensor product of the form 

Px ® • • • <8> pt 

such that pi, Prank(o-) are extensions of representations (with nontrivial central char- 
acters) of H 1 , ff rank (°") respectively, and the rest of the p/s are trivial. The first 
requirement imposes open conditions on the subset of rankable representations of a given 
rank, whereas the second requirement imposes a closed condition. 

Definition 5.3.3. Let tt be any unitary representation of G. tt is called "pure-rank" if 
its restriction to Np is a direct integral of rankable representations of a fixed rank. The 
common rank of these rankable representations is called the rank of tt. 

Although Theorem 15.3.21 is slightly stronger than Corollary 15.3.41 below, we would like 
to state it in order to clarify the analogy between our new theory and the older one. 

Corollary 5.3.4. Let tt be an irreducible representation of G. Then tt is pure-rank. 

5.4. Proof of Theorem 15.3. 11 In this section we prove Theorem 15.3.11 Without loss 
of generality we can assume that condition (J5.1j) holds. Theorem 15.3.11 is proved by in- 
duction on ht(iVr). Let Np be described as in (CTHjl . By Howe-Moore's theorem [HMJ , 
when ht(iVr) = 1 there is nothing to prove. Now let ht(iV r ) > 1. Let P, L, M and N be 
as in section |5~T1 Let Pi, Mi be as in Proposition 15 . 1 . ll The H-tower subgroup which 
is associated to M is N 2 = Np/N and it lies within M x . But ht(N 2 ) = ht(JV r ) - 1, and 
therefore Theorem 15.3.11 holds for M\. 

Without loss of generality, we can assume tt has no nonzero G-fixed vectors. Let 
(Tj's be the representations which appear in the decomposition of tt using Lemma [5.2.11 
By a straightforward application of Mackey's subgroup theorem (see Theorem 15.2.3)) . or 
even more directly (with only little difficulty) by writing the definition of the induced 
representation IndjCTj explicitly, one can see that the representation 

ReSp IndjOi 

is supported on representations of the form 

Res^of x e P 

where of is defined as in (|5.4j) ; i.e. 

(5.5) for any y G J, erf (y) = a-^x^yx). 

Note that J is a normal subgroup of P because it contains [P, P] . 

By ([5.3)1 . Res^of is unitarily equivalent to vf ® p* where uf and pf are defined similar 
to (|5.5|) . Theorem l5.3. ll follows from Lemma lo . 2 . 2l applied to pf, the fact that by induction 
hypothesis uf is supported on rankable representations of N 2 = Np/N, and Definition 
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5.5. Proof of Theorem 15. 3. 21 We will now proceed towards proving Theorem 15. 3. 21 
Without loss of generality, we can assume that condition (|5.1|) holds. Let P, L, N and 
M be as in section IB~T1 Let Pi, Mi be as in Proposition 15.1.11 The main idea behind 
the proof is that the Kirillov orbits of representations of different rank have different 
dimensions. We will apply some basic Kirillov theory. 

Recall from section IS~B1 that G, g, $jp are as in section H~2l and g satisfies condition (j3.1j) . 
Let S, S + and be as in section rOl too. Let ft be the highest root of g (see section 
II. 2j) and ft be a simple restricted root such that (ft, (3) = 1. Recall that 

(5-6) P m = L m N m 

is a standard parabolic subgroup of G associated to {ft} (see section E2J) . Let N-p be the 
H-tower subgroup of G and suppose nr is its Lie algebra. Let N^py be as in equation 
(J5.6J) and suppose nypy is its Lie algebra. We consider nr and nypy as subalgebras of gp. 
Recall that for any 7 G £, (0f) 7 denotes the restricted root subspace of gF associated to 

7. Our next aim is to define a subgroup of Nr in case ht(iVr) > 1. 

Let Lr be as in equation (|3.4jl . with Lie algebra h where h C gp, and 

£ L = {7 G S I (g F ) 7 C l r }. 

Define 

(5-7) 4=(& (g F ) 7 

where 

(5.8) S r = {7 G £ + | (g F ) 7 Cn r n n m and 7 - /? £ 

One can see that rip is a Lie subalgebra of gp. To see this, assume 7,7' G Sr and 
7 + 7' G S + . Then since both nr and n^} are Lie algebras, (gp) 7+ y C tip fl K{py- 
Moreover, if 7 + 7' — /5 G S L , then either 7 G X L or 7' G S L which is a contradiction. 
Consequently 7 + 7' G Sr- 

The group iVp is defined as the subgroup of iV r fl Nspy with Lie algebra rip. (It is 
worth mentioning that iVf is a proper subgroup of Nr fl Nspy if and only if T contains 
an element which is not orthogonal to ft in S.) 



Lemma 5.5.1. Letht(Nr) > 1 and L^y be as in equation \5.b\) . Let M spy = [Lspy, Lspy], 
the commutator subgroup of Lspy. Then M^y normalizes iVp\ 

Proof. It suffices to show that for any 7^^, if 7 + G £ (respectively 7 — ft G S) then 
7 + ft G SV (respectively 7 — ft G Sr). Note that J — ft cannot be zero. 

Assume 7 G Sr and 7 + ft G S. Since (gp) 7 C tl{/3}, we have (gF) 7+ ^ C nypy too. Simi- 
larly, since (g F ) 7 C n r , 7 + /? ^ S L and (7 + /?) — /? = 7 ^ S L which imply that 7 + /? G S r . 

Next assume 7 G Sr and 7 — /3 G S. Note that 7 — ft G E implies 7 — ft G S + since 
7 G S + . Again since (qw) 1 C we have (gp) 7 _ /3 G tt{j8}. Moreover, by the definition of 
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tip, 1-/3 £ S L and therefore (0f) 7 _^ C n r . Finally, if (7 - (3) - (3 G £ L , then 7 = 2/5 + 71 

where 71 G S L . This means that (7,/?) = 2, i.e. 7 is the highest root. Since any simple 
restricted root appears in the highest root with a positive coefficient, it follows that E B 
consists of (3 and the elements of V . Consequently, there is only one simple restricted 
root outside T, which implies that ht(iV r ) = 1, which contradicts our assumption. 

□ 



Proposition 5.5.2. Let G be as in Theoreom \5.3.2i be the H-tower subgroup ofG and 
ht(iVr) > 1- Let ivf be defined as in equation \5. 7\ ). Then the restriction of a rankable 
representation a of rank k of to is a direct integral of irreducible representations of 
whose attached coadjoint orbits have the same dimension equal to 2(ni + - • - + 72fc — c), 
where c is the codimension of in Np, and n, ? s are defined as in Corollary \4- 2. 3[ 



Proof. Throughout the proof we assume F = K for simplicity. The proof for other local 
fields is essentially similar. 

We will analyze 

Res^g(pi ® • • • ® p k ) 

with the pi's as in Definition 14.1.11 It is easy to see that iVp D H 2 ■ ■ ■ H h W, so when 
j > 1 the restriction of pj to is irreducible. It remains to understand the restriction 
of pi to Nfl. The group is a direct product of a Heisenberg group of dimension 

2(rzi — c) + 1 and a c-dimensional abelian group whose Lie algebra corresponds to the 
direct sum of restricted root spaces (fla)^ , such that (7,/?) > but 7 ^ St- To see why 
these restricted root spaces form an isotropic subspace of the Heisenberg nilradical of 0r, 
suppose (3 — 71, ft — 72 are given such that for any i G {1, 2}, (7^ ft) > but ji St- If 
(/3-7i) + (/3-7 2 ) = ft, then (3 = 71 + 72. But for any i G {l,2}, 7i = ft + i[ where 7^ G S L 
or 7 4 ' = 0. Therefore (3 = 2/?+7^+7 2 , which implies that consists of (3 and the elements 
of r. (See the proof of Lemma 15.5. 11 ) Consequently ht(iVr) = 1, which is a contradiction. 

Let the decomposition of H 1 n as a direct product be x M. c , where is the 
2(ni — c) + 1-dimensional Heisenberg group and M c is the abelian factor. We will denote 
the irreducible representation of H@ with central character xi by pf. 

Lemma 5.5.3. Under the foregoing assumptions, 



"r 



where each ip s {t), given by ip a (t) = e ls ^ /or some s G M c * , £/ie vector-space dual of~EL c , is 
a unitary character ofR c . 

Proof. Clearly 

B£8 WnN? Pl = Res SxR C Re 4iPi = Res^ xRc pi. 
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The space H p of any representation p of the Heisenberg group H 1 = H ni introduced in 
section 2.2 can be written as 

Hp = L 2 (t)) = L 2 (Span K {n, ■ ■ ■ ,Y C } ® Span R {F c+1 , . . . ,Y ni }) = 

L 2 (M c )®L 2 (Span M {F c+1 ,...,F ni })^ / L 2 s dp(s) 

where each L 2 is equal to L 2 (Span R {Y" c+1 , . . . , Y ni }) on which M. c acts via the character 
Xs( x ) — e ls( - x \ In fact L 2 S is a representation of H 13 x R c . Lemma f5. 5. 31 is proved. □ 

Next we show that W C Z(N^). To see this, take a restricted root space (0r) / 3_ 7 
which lies inside M. c . It suffices to show that for any 7' G Sr, {(3 — 7) + 7' ^ S. But we 
know that 7 = /? + 71 where 71 G S L or 71 = 0. Therefore (j3 — 7) + 7' = /5 — /? — 71 + 7'. 
Consequently, if (/3 — 7) + 7' G £, then either 7' 6 S L or 7' — (3 = 71 G S L . However, 
none of these is possible for 7' by the definition of Sr in ()5.8|) . 

We have shown that M c C Z(N^) and the action of IR C on each L 2 is by a distinct 
character ip s (see Lemma l5.5.3j) . Hence the restriction of p\ to N@ breaks into a direct 
integral of a c-parameter family of irreducible representations. Consequently the same 
thing happens to any rankable representation 

cr = pi <g> • • • <g> p k . 

By Theorem 2.5.1 of |CCj and Lemma \A. 2 . 21 applied to iVj? which is a semidirect product 
of H" x M c and A^2, it follows that the projection of the coadjoint orbit O* onto the Lie 
algebra of is foliated by subvarieties of codimension 2c, which are indeed coadjoint 
orbits of the constituents of the rankable representations in the restriction of a to N^. 
Proposition 15.5.21 is proved. □ 

We will now concentrate on finishing the proof of Theorem 15.3.21 Theorem 15.3.21 is 
proved by induction on the height ht(iVr) of the H-tower group N-p. If ht(iVr) = 1 then 
there is nothing to prove. Let ht(iV r ) > I. Then the H-tower subgroup of Mi is equal to 
A^2, where 

N 2 = N r /N. 

Clearly H(N 2 ) = ht(JV r ) - 1. 

Without loss of generality we can assume 7r has no nonzero G-invariant vectors. Con- 
sider the decomposition of 7r given in (J5.2j) . Applying induction hypothesis to Mi, which 
contains the H-tower subgroup N 2 , we can refine this decomposition by expressing each 
Vi as a direct sum of its Mx-invariant pure-rank parts (where the rank for a representation 
of Mi is naturally defined with respect to N 2 ). Conseqently, as a representation of Mi, 

Vi = Vi,o ffi n,i ffi • • • ffi ^i,ht(iv 2 )- 

Here v^j denotes the part of supported on rankable representations of N 2 of rank j. 
(Note that some of the z^jj's may be trivial, but this fact does not affect our proof.) 
Therefore we have 

CTi\ Pl = {Vifl ® Pi) © " • " © (^,ht(JV 2 ) ® Pi)- 

Let 7i be the Hilbert space of the representation cij. For any j G {0, 1, 2, ht(A^ 2 )}, let 
Tij be the subspace of TC which corresponds to ® p%- Our next task is to prove that 
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each Tij is in fact invariant under <7j(J). (See Lemma [5.2. 11 ) This follows from Lemma 
15.2.41 once we prove the following lemma. 

Lemma 5.5.4. Let 77 = v^j ® pi and 7/ = Vij> <S> p% where j 7^ f . Let a £ J. Then 

Hom Pl (r/W) = {0} 
where rf is a representation of P\ defined on Hj defined as in \5.$ ; i.e. 

r] a (x) = r](a~ 1 xa) for all x G Pi. 
Proof. We actually prove more; i.e. that 
(5.9) Rom Nr (Resgrj a , Res^r/) = {0}. 

We claim that Res^,^ a is a direct integral supported on rankable representations of 
of rank j + l. The claim implies ()5.9|) because if j 7^ j', then it implies that Res^^ and 
Res^r]' are supported on disjoint subsets of Nr. We prove this claim below. 

By Lemma f5. 2. 21 pi is supported on rankable representations of N-p of rank one. Next 
we show that i/? - is supported on rankable representations of N 2 = Np/N of rank j. 

As we know, G is a central extension of Gp. Suppose this extension is represented as 
in ([IID . 

First let a G N. Then a G N? and consequently Kes^rj' 1 is unitarily equivalent to 
Resj^r?. Since J C P and any element of P is a product of an element of N and an 
element of p _1 (Lf), it follows that it suffices to assume that p(a) G Lf. 

It follows from p(a) G Lp that p(a)Nlfp(a^ 1 ) = Mp. The group p{Pr) is the F-points 
of the F-parabolic Pp' of G. Now P m = Pr' fl M is an F-parabolic of M. Since 
p(a) G Lp, p(a)P m p(a~ 1 ) is another F-parabolic of M, and therefore by |Bor[ Theorem 
20.9] these two parabolics are conjugate under an element p(b) of Mp where b G Mi (re- 
call from Proposition 15 . 1 . il that p(Mi) D Mp). This means that p(6a)P m p(6a) _1 = P m . 
If U m is the unipotent radical of P m , then p(6a)U m p(6a) _1 = U m , which implies that 
baN^b" 1 = N 2 . 

Let c = ba. Then c G P. Let n 2 C rip be the Lie algebra of N 2 . Consider Ad*(c) as a 
linear map from the dual of the Lie algebra of P to itself. Obviously Ad*(c)(n 2 ) = n 2 . 
Let r be an irreducible unitary representation of N 2 and let O* be the coadjoint orbit 
associated to r. Then the coadjoint orbit associated to r c is Ad*(c)(0*). This fact fol- 
lows for instance from |Du| III 11]. A short proof of this fact can be obtained by an 
adaptation of the proof of Lemma 15.5.61 

For simplicity let v = u^j for fixed The representation Resj^V is supported on 
rankable representations of N 2 whose associated coadjoint orbits have dimension 

n 2 H V n j+ i. 

(See Corollary 14.2.31 for the definition of n»'s.) 
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Let u a , u b , v c be representations of M\ defined as in (jPD - (Note that aM^' 1 = M 1 
because Pi is normal in p _1 (P]j-).) Since b G M 1; any representation 9 of Mi is obviously 
unitarily equivalent to 9 h . Therefore v c is unitarily equivalent to v a . Since the action of 
Ad*(c) on rig is linear, it does not change the dimension of coadjoint orbits, and therefore 
Resj^V c is supported on unitary representations of N 2 whose associated coadjoint orbits 
have dimension n 2 + • • ■ + nj+%. 

Since v c is a unitary representation of Mi, by Theorem 15.3.11 all representations in 
the support of its restriction to A" 2 should be rankable. Therefore Res^V c is supported 
on rankable representations of N 2 of rank j. Now v a is unitarily equivalent to v c as a 
representation of Mi, and hence as a representation of N%. Therefore v a is also supported 
on rankable representations of N 2 of rank j. Definition 14. 1 . II completes the proof of our 
claim. The proof of Lemma 15.5.41 is complete. 

□ 

As mentioned before, Lemma f5.5.4l implies that each of the subspaces Tij (which cor- 
responds to the representation Uij ® pi of Pi) is invariant under o~i{J). 

Lemma 5.5.5. There is a P -invariant direct sum decomposition of tt such as 

7V\p = 7Ti © • • • © 7r ht(A r r ) 

where each Hi is of pure N^-rank i. 

Proof. The invariance of the spaces Tij under <7j(J) means that one can actually decom- 
pose <Tj as a direct sum 

&i = © &i,2 © • • ■ © CTi,H(N T ) 

such that each o~ij is a representation of J and 

< J i,j\P 1 = v i,i-\ ® Pi- 

Now an application of Mackey's subgroup theorem (see Theorem I5.2.3j) immediately 
implies that the representation 

Res^ r IndjCTij 

is supported on rankable representations of Ar of rank j. (See the argument of Theorem 
15.3.11 and Lemma 15.5.41 ) Lemma [5.5.51 is proved. □ 

Recall from Lemma [5.5.11 that the group A^p is normalized by M^. A version of the 
following lemma was mentioned in the proof of Lemma 15.5.41 

Lemma 5.5.6. Let r be a unitary representation of and let g G M^}. Let t 9 be 
defined as in \5.4\ i- Then the coadjoint orbits associated to r and t 9 (in the sense of 
Kirillov's orbital theory) have the same dimension. 

Proof. The proof follows immediately from the fact that the coadjoint orbit attached to 
t 9 is Ad*(g)(0*), which follows from |Dul III 11]. We would like to give a short proof of 
this fact for the reader's convenience. Let rip be the Lie algebra of Aj?. Fix an additive 
character x of F as done in |Mol| §4]. (When F = M or C, x(t) = e lRe( ^ where Re(t) 
means the real part of t, and when F is p-adic, \ will be an unramified character given by 
Tate.) Let the coajoint orbit associated to r be O* C rip. By Kirillov's orbital theory we 
know that r is constructed as follows. One chooses an arbitrary element A G O* and a 
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maximal subalgebra q of rip subordinate to A, which exponentiates to a closed subgroup 
Q of N$. Then 

iV 3 

r = Ino V(x°Aolog). 
For any g G M^}, let (x°Ao log) 5 be defined as in (15 .4|) . Then 

r 5 = Ind^^x o A o log) 5 = IndJ^x ° Ad*(g)(\) o log). 

Since Ad(g)(c\) is a maximal subalgebra subordinate to Ad*(g)(X), the coadjoint orbit 
attached to t 9 is 

Ad*(g)(0*). 

Since the action of Ad*(g) is linear on rip, it does not change the dimension of the 
coadjoint orbit. □ 

To prove Theorem 15.3.21 we show that each of the components 7Tj given in Lemma 
15.5.51 is G-invariant as well. To this end, we first prove the following lemma. 

Lemma 5.5.7. The P -invariant decomposition in Lemma \5. 5. .51 is preserved by the action 
of M{p}, where 

M m = l L {/3}>£{/3}] 

and L{p} is the Levi component of the standard parabolic P^} with (3 as in Proposition 
Proof. For any 1 < j < ht(iVp), the representation 

p 

is a direct integral of representations which correspond to coadjoint orbits of dimension 
2(ni + • • • + rij — c). However, ni, ...,n ht ^ Nr ^ > 0. Therefore, if we define 7r" (for any 
a G M{/3}) as in (|5.4j) . then Lemma 15.5.61 implies that for any j' ^ j, the dimension of 
the coadjoint orbits associated to the irreducible representations of iVp in the support of 
Res^ r 7r" is different from the dimension of the coadjoint orbits assoiated to the irreducible 
representations of iVp in the support of Kes^Tij/. This means that 

Hom^Res^TT^Res^TTj/) = {0}. 
Now we apply Lemma 15.2.41 with K = [P{/3}, P{p}], K' = N^, a = ir, a% = Hi and 

°" 2 = ® 1Xi - 

It follows that each is invariant under the action of M^. Lemma [5.5.71 is proved. 

□ 

To finish the proof of Theorem 15.3.21 we note that the parabolic subgroup P in G is 
maximal, therefore the group generated by M{py and P will be equal to G. (This follows 
from the Bruhat-Tits decomposition.) The decomposition of tt given in Lemma f5. 5. 51 is 
preserved by both P and M{@y, and hence by the group generated by them. Therefore 
the decomposition of Lemma [5.5.51 is G-invariant. 

Remark. Let G be as in Theorem 15.3.11 such that the central extension identifying 
G is as in (jl.ljl . Let iVp be the H-tower subgroup of the group G. Let ir be a unitary 
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representation of G of pure rank k, where k < ht(iVr). Consider the subgroup Ak of 
G defined as follows. Ak = p _1 (Ap), where A' is an F-torus inside the maximal split 
F-torus A whose Lie algebra is spanned by the coroots H^, Hs . Let r be a rankable 
representation of Nr of rank k. The stabilizer S T of r inside Ak is a finite subgroup of 
Afc. Moreover, under the action of Ak there are only a finite number of orbits of rankable 
representations of rank k. By Mackey theory, the restriction of ix to Ak K Nr is a direct 
integral of representations of the form 

(5.10) hd£3§>T 

where a T is irreducible and a T \ Nv = n T r for some n T G {1, 2, 3, oo}. Moreover, by 

Frobenius reciprocity, a T will be a subrepresentation of Ind^ r * JVr T. Since S T is a finite 
group, there are only a finite number of possibilities for a T . Since the number of orbits 
(and hence stabilizers) of rankable representations of rank k under the action of Ak is 
finite, the number of representations of the form ()5.10|) is finite as well. This implies the 
following result. 

Proposition 5.5.8. Let n be a unitary representation of G of pure rank k. Then there 
exists a finite family {r 1; ■■-,r t \ of irreducible representations of A^ x Nr, independent of 
Ti, such that 

n\A k KN T = n X Ti H \-n t T t 

where rii G {1, 2, 3, oo} for each i. 

6. Relation with the old theory 

6.1. Outline of the old theory. In this section we show how the notion of rank defined 
in the past sections relates to the existing theory for classical groups. To this end, we 
show that for the real forms of classical groups, the two notions of rank (the one defined 
in |Lilj and the one defined in Definition 15. 3. 3|) are equivalent. Here we give a brief out- 
line of the old theory. In classical cases, rank of a representation of the real semisimple 
group G is defined in terms of its restriction to the centers of nilradicals of maximal 
parabolic subgroups. One can characterize each of these parabolics with a node in the 
Dynkin diagram of the restricted root system in a natural way. It turns out that there is 
a (not necessarily unique) standard parabolic which provides the most refined informa- 
tion about the rank. We will devote this section to exhibiting the coincidence of the two 
notions of rank on this parabolic. Really the main idea is some slight modification of 
the fact that the nilradical of the rank parabolic subalgebra contains a maximal isotropic 
subspace of each of the Heisenberg algebras in the H-tower Nr. Our presentation of the 
results follows the notation of older literature |Hwlj . |Lilj . |Scaj . 

The notation used in this section is chosen independent of other sections in order to 
simplify matters and be more coherent with older works. For simplicity we only consider 
the case F = R. The general case is essentially the same and will only be more technical. 
It is more convenient to consider classical groups of different types (in the sense of |Hw5j ) 



separately. Here we quickly review the definition of classical groups of types I and II over 
a local field, but later we will retain our assumption that F = R. 



Let F be a local field, D a division algebra over F with an involution, and V a left vector 
space over D of dimension n. A classical group G is said to be of type II if G = GLe>(V). 



A NOTION OF RANK FOR UNITARY REPRESENTATIONS 



29 



From now on, by (•, •) we mean a Hermitian or skew-Hermitian sesquilinear form (•, •) on 
V. A classical group G of type I is the connected component of identity of the stabilizer 
subgroup of (•, •) inside GL D (V). The real groups of type I which are of our interest 
here, i.e. those which satisfy the assumptions of Proposition 13.1.11 correspond to the 
cases where F = R and D = 1, C or H with their usual involutions. 

6.2. Groups of type I. A typical maximal parabolic of these groups can be described 
as follows. Take a maximal polarization inside V, i.e. a maximal set of vectors 

jei, . . . , e r , e l , . . . , e r y 

in V which satisfy 

(e^ej) = (e*,e*) = 

In fact r is equal to the split rank of G. For any k let 

X k = Span D {ei, . . . ,e fc } , X* k = Span {e*, . . . , e* k } 

and define V k to be X k Q)X k . Let P k be the subgroup of G that consists of elements which 
leave the subspace X k invariant. P k is a parabolic subgroup and the Levi decomposition 
of P k looks like 

(6.1) P k = GL D {X* k )G{V^)N k 

where by G(V k x ) we mean the stabilizer of (■, ■) as a form on V k . Here N k is the unipo- 
tent radical of P k . 

P r is the parabolic which provides the most refined information about the rank (in the 
sense of [HwT] . |LTT] ) . 

Definition 6.2.1. Let r be the split rank of G. The parabolic P r or its Lie algebra are 
called the rank parabolic. 

The unipotent radical N k is a two-step nilpotent simply connected Lie group and there- 
fore it can be identified with its Lie algebra via the exponential map. From now on, we 
think of any N k through this identification, and although slightly ambiguous, we use the 
same notation for its Lie subgroups and their Lie algebras. This is done in order to avoid 
complicated notation and to keep the presentation as close to the style used in the papers 
of Howe and Li. We will make it clear whether or not we are using a Lie group or a Lie 
algerba wherever necessary. 

As in |Lilj , we have the following exact sequence of Lie algebras: 
— ZN k — N k — RomniV^, XI) — 
where ZN k is the center of iV^. ZN k is isomorphic to 

Hom£ v (X fc ,X*) 

where Hom^ v (Xfc, XI) is the F-subspace of elements T of Homr,{X kl XI) satisfying 
Vi,j E {1,2,..., k}, (Tei,ej) + (e^Tej) = 0. 
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Thus as an F— vector space, the Lie algebra N k can be expressed as 

(6.2) N k = Hom D (^, X*) © Hom£ v (X fc , X* k ). 

The isomorphism of RomoiV^, X^) into the Lie algebra Nk can be depicted as 

(6.3) ~:Rom D (V k ± ,X* k ) ^ Rom D (V,V) 

T !-»• f 

where T is defined as follows: 

Tv = Tv for v G V k 
fv = for i)6 X* 
Tv = T l v for v G X fc . 

Here T* G Horn^X^, V^) is defined uniquely by 

VveV,, 1 , x G X k {Tv, x) + (v, T*x) = 0. 

It turns out that the Heisenberg parabolic P of G is P^, where k\ = 2 for G = SO Piq 
and fci = 1 for all other classical cases under consideration. Let P = LN be the Levi 
decomposition of P. Let M be the appropriate simple isotropic factor of [L, L\\ i.e. we 
drop the redundant factor of [L, L] which, in (j6.1J) . corresponds to GL^ (Xj^ ) . Let m be 
the Lie algebra of M. For any k define 

Y k = Span{e fc , ...,e r } , Y k * = Span{e£, . . . , e*}. 

The center of the nilradical of the rank parabolic of m is identical to 

HomS v (F fel+1 ,r; i+1 ). 

The Lie algebra Hom 1 ^ v (F fcl+1 , Y k * +1 ) acts on the Lie algebra N kl through the adjoint 
action of m. By Theorem 13.1.11 this action will be trivial on ZN kl . The following simple 
lemma describes this action more explicitly. 

Lemma 6.2.2. Let X G Homg v (y fc+1 , Y k * +1 ). Let Y G Hom D (F fc ± , X£). Then the adjoint 
action of m on n = N k is described as 

ad x (f) = -(FX) 

where YX is defined as in (QT3J). 

Remark. Note that we think of —YX as an element of Hom£ ) (V r fc " L , X k ) which is zero on 

The restriction of (•, •) to Vf- is a definite form. Without loss of generality, we may 
assume that the form is positive definite. Let {/i, . . . , f n -2r} be an orthonormal basis 
for V c = V^~\ consequently V c = Span D {/i, . . . , /„_2r}- One can see that H.om D (V k L , X k ) 
is equal to 

(6.4) Rom D (Y k+1 ,X*) ®Kom D (Y* +1 , X* k ) Kom D (V c , X*). 

We consider the direct sum decomposition (|6.4j) inside N k (see (16. 2|) ) We observe that: 
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Lemma 6.2.3. In the direct sum decomposition \6.4\) , the third summand commutes 
with the first two summands, and the adjoint action of Hom^ v (Y/ c+1 , Y k+1 ) on the third 
summand is trivial. 

Now take k — k\. Then the direct sum 

(6.5) Rom D (Y kl+1 ,X* ki ) © Rom D (Y* i+1 , X fc *J © Hor<(X fcl , X* kl ) 

is a Lie subalgebra of N kl , and also a Heisenberg algebra with a polarization given 
by the first two summands in (jfi.5j) . We denote the Lie algebra in (16. 5 j) (and also its 
corresponding Lie group) by N kl ■ The Lie bracket when restricted to the polarization is 
described as follows. 

Lemma 6.2.4. Let X G Hom D (F fcl+1 , X*J and Y G Rom D (Y* i+1 , X*J. Then [X,Y] is 
an element of Hom^ v (Xfc i; X^J given by 

[X,Y] =XY t -YX t 

where X 1 G Hom^X^, Y k * +1 ) and Y l G Honi/^X^, Y kl+ \) are uniquely determined as 
follows 

V i < ki and V j > ki, (X t e i , e 3 ) + {e i: Xe 3 ) = 

V i < h and V j > k u (Y t e i , e*) + (a, Ye*) = 0. 

Proof. Follows immediately from [X, Y] = XY — YX where X and Y are defined as in 

The adjoint action of Hom^ v (Y" fcl+1 , Y k +1 ) on N kl is given by Lemma 16.2.21 This ac- 
tion normalizes N kl and takes Hom£>(Y fc * +1 , X k ) to Hom£)(Y'fc 1+ i, X k ). 

At this point we come back to nilpotent groups and their representations. Consider 
an irreducible representation pi of the Heisenberg group N kl with (nontrivial) central 
character xi- From the orthogonal decomposition obtained in (|6.5|) it follows that the 
restriction of p\ to the group N kl decomposes into a direct integral of representations of 
this latter Heisenberg group with the same central character. We study the restriction 
of a rankable representation of rank one of the H-tower unipotent radical of G to its 
subgroup 

w kl xHoms v (n 1+ i,y; 1+1 ). 

This restriction is a direct integral of representations of the latter group obtained by 
extending the irreducible representation of N kl with central character xi to N kl x 
Hom^ v (Yfc 1+ i, Y k +1 ) as suggested by Proposition 12.3.11 This is because Lemma T6.2.3I 
implies that as subspaces of N kl , 

[HomS v (n i+ i,n* 1+ i),Hom D (K,X* i )] = {0} 

and the Weil representation is functorial. (See (1.15) of [Hwl for a precise meaning of 
functoriality.) 

Since Hom^ v (Yi, Y*) is an abelian Lie group, any unitary representation of this group 
can be described as a direct integral of unitary characters. Hom^ v (lx, Y*) is isomorphic 
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to some W; so its group of unitary characters can be naturally identified to Hom^ v (y L *, Y\) 
via the F-bilinear form 

(6.6) p(A,B) = \x{AB). 

Definition 6.2.5. The rank of a character o/Hom£ V (Yi, Y*) is the ¥-rank of the element 
in Hom 1 ^ v (Y 1 *, Yi) which corresponds to it via the bilinear form in 46'. 6)) . 

Definition 6.2.6. (See |Hwl| . |L"iT] ) Let tt be a unitary representation of G. n is said to 
have rank k if and only if TT\u m i g v (Y 1 ,Y*) decomposes into a direct integral of characters of 
rank equal to k. 

The following proposition is a key result of this section. 
Proposition 6.2.7. The restriction of a rankable representation of rank one to 

HomS v (Yi,l?) 
is supported on characters whose rank is equal to k\. 

Proof. The polarization for the group N kl has the structure of a D-vector space. There- 
fore, similar to ()2.5j) . we can realize the representation p\ of N kl on 

L 2 (Hom D (y; i+1 ,^)) 

and then extend it to Hom^ v (Yfc 1+ i, Y k+1 ). In Lemma 16.2.81 below, we denote elements 
of the Lie algebras by X,Y,... and the corresponding elements in the Lie groups by 



e x ,e y ,.... 



Lemma 6.2.8. The action of the extension of p\ is described as below. 

(a) For any X G Hom D (Y fel+1 , X*J, Y G Rom D (Y* +1 , X* ki ) 

(pi(e x )f)(Y) = xi(e^)f(Y) 

(b) For any X G Rom^(Y kl+u Y* i+1 ),Y e Eom D (Y* i+1 , X* k J 

{ Pl {e x )f){Y) = X i{e^ YX] )f{Y) 

(c) For any X G Rom™(X kl , X* kl ), Y G Rom D (Y k * 1+1 , X* k J 

{pi{e x )f){Y) = Xl {e x )f{Y). 

Proof. This is an almost immediate consequence of the Schrodinger model for the real- 
ization of Weil representation. See [Hw6] . □ 

Let X G Hom£ v (Y fcl+1 , Y fc * i+1 ) and Y G Hom D (Y fc * +1 , X^). For all % < k x and j > k u 

(e„ (YXfa) = -(YXe^e,) = (Xe^Y^i) 

= -(e j ,XY t e i ) = (e 1 ,-XY t e i ) 

which implies that (YX) 1 = —XY l . Thus the equation in part (b) of Lemma f6.2.8l can 
be simplified as 

(p 1 (X)f)(Y) = Xl (e- YXYt )f(Y). 

We would like to have a single formula instead of parts (a), (b) and (c) of Lemma 
16.2.81 To this end, we define the linear operator S = S(Y) such that 
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by 

Se* = e* if i < k\ 
Se* = Ye* if i > k x . 

We have the following lemma. 

Lemma 6.2.9. Let X E Hom^ v (F 1 , Y*). Let f be a function such that 

feL 2 (Kom D (Y k * i+1 ,X* ki )). 

THgti 

Pl {e x )f{Y) = X i{e- sxst )f{Y) 
where S* is defined as in Lemma \b\2.J\ 

Proof. Applying (e i: Se*) + (.S'ej, e*) = 0, one can see that for any i < k\ 

S t Ci = Ci + Y t e<i 
and thus for any X G Honi D (Y" fcl+1 , X k ) 

-SXS'ei = -SX(-ei + Y'a) = 
{-3){-X t )e i -3XY t e i = 
YX t e i -XY t e i = [Y,X] ei 

which proves —SXS 1 = [Y,X]. 

For any X G Hom i g v (Y fcl+ i, Y k \ +1 ) we have 

-SXS'ei = -SX(- ei + Y'a) = -SXY'ei = —YXY t e i 
which proves -SXS 1 = —YXY 1 . 

Finally, when X e Hom£ v (X fel , X*J 

-SXS'ei = -SX(- ei + Y*ei) = SXa = Xe { 

which proves —SXS 1 = X. This completes the proof. □ 

To complete the proof of Proposition 16.2.71 note that via the identification described 
in (jfi.fijl . the character xi eX ) = Xi{ e ~ S XS ) corresponds to an element 

(6.7) — ^S t VS aeiR-{0} 

h 

where 

V : X* ki » X kl 

is defined as 

Ve\ = (-l) l+1 e* kl _ l+1 for any 1 < I < k x . 

It is easy to see that ()6.7|) is an element of Hom^ v (Y 1 *, Y\) of rank hi, even when its 
domain is restricted to X kl . □ 

The following theorem shows that in groups of type I, the two notions of rank are 
essentially the same. 

Theorem 6.2.10. Let it be an irreducible representation of a classical group G of type 
I. Let Nr be the H-tower subgroup of G (see \3.4\))- 
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• Assume that the rank of tt in the sense of Definition ^. 2. b\ is less than ht(iVr) x k±. 
Then tt has rank k in the sense of Definition \b.3."<A if and only if tt has rank kk\ 
in the sense of Definition ^. 2~b\ 

• If the rank of tt in the sense of Definition \5. is equal to ht(iVr) ; then rank of 
tt in the sense of Definition ^. 2~R is ht(iVr) x k± or higher. 

Proof. Since tt is supported on rankable representations, the only thing we have to show 
is that a rankable representation of rank k (in the sense of Definition I4.1.1|) . when re- 
stricted to Hom 1 ^ v (Y" 1 *, Yi), decomposes as a direct integral of characters of rank kk±. By 
Proposition 16.2.71 this is true when k = 1. 

Next consider a rankable representation p of rank k > 1, say 

p = Pi® ■ ■ ■ ® Pk- 

Elementary properties of tensor product imply that the restriction of p to Hom 1 ^ v (Y" 1 , Y*) 
is a direct integral of characters of the form 

01 ■ 02 

where the characters 0i and 02 are constituents of the direct integral decomposition of p\ 
and pi® ■ ■ ■ ® Pk, when restricted to Hom^ v (Yi, Y*), respectively. But if 0j {i G {1, 2}) 
corresponds to A4 G Rom^iY*, Yi) via then X • <p 2 corresponds to A% + A 2 . Since 

P2 ® ■ ■ ■ ® Pk and therefore any possible 02 is a trivial representation when restricted 
to Nfai any possible A 2 is really an element of Hom I ^ v (F A .*_ | _ 1 , l^i+i) which is extended 
trivially on to Y£. However, at the end of the proof of Proposition 16 . 2 . 71 it was shown 
that any such character (f>i corresponds to some element of Hom^ v (Y 1 *, Y{) which is of 
rank k\ even when its domain is restricted to X% . It is now easy to show that we have 

rank(Ai + A 2 ) = rank(Ai) + rank(y42) — k\ + rank(A2). 

An induction on k completes the proof. □ 

Example. Let G = SO (6, 6). Let tt be an irreducible unitary representation of G. Then 
the rank of tt in the sense of Definit ion 16 . 2 . 61 can be 0, 2, 4 or 6. The rank of tt in the sense 
of Definition 15.3.31 can be 0, 1 or 2. The following chart shows how the ranks correspond 
to each other: 



Definition 16.2.61 


Definition |5.3.3| 








2 


1 


4 


2 


6 


2 



Now let G = SO(5, 11). Then we have a similar chart for the rank of tt. 



Definition 16.2.61 


Definition 15.3.31 








2 


1 


4 


2 



Therefore the correspondence of ranks may or may not be one to one. It is an easy 
exercise to determine in which cases the correspondence is actually one to one. 
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6.3. SLi + i(R). Let G = SLi +1 (R), the group of linear transformations on the / + 1- 
dimensional vector space V with a fixed basis 

{e 1 , . . .,e l+1 }. 

Set r = L^jrJ • F° r an y k let Pk be the maximal parabolic of G which is represented by 
matrices of the form 

A B 
C 

where 

AeGL k (R) , CeGL,+i_ fc (M) , and B E M fcx(J+1 _ fc) (R). 

Therefore 

P k = S(GL k (W) x GL l+1 . k (R)) • N k 

where 

N k = Hom(Span({e fc+ i, . . . , ej+i}), Span({ei, . . . , e k })). 

The parabolic of G which gives the most refined rank is P r , and henceforth we focus our 
attention to P r . Let 

X k = Span({e 2 , . . . ,e k }) , Y k = Span({e fc , . . . , ej}). 

The Heisenberg parabolic subgroup of G is Pi n Pi, and a polarization of the Lie algebra 
of its unipotent radical iV is a direct sum of 

Hom(X r , Rei) © Hom(Re i+1 , Y r+1 ) 

and 

Hom(y r+ i, Rei) © Hom(Re m , X r ). 

The second summand lies inside the nilradical of the Lie algebra of P r . Its center is 
isomorphic to 

Hom(Re J+ i,Rei). 

As before we are interested in description of the restriction of a representation p\ of N 
to 

Hom(Span({e r+ i, . . . , e i+1 }), Span({ei, . . . , e r })). 
We identify the dual of 

Hom(Span({e r+ i, . . . , e J+ i}), Span({ei, . . . , e r })) 
with itself via the bilinear form 

(3(X,Y) = tr(X t Y). 

The rank of a unitary character of 

Hom(Span({e r+ i, . . . , e i+1 }), Span({ei, . . . , e r })) 

is defined to be the rank of the linear transformation which corresponds to it via the 
bilinear form (3. 

We write any 

Y E Hom(X r , Rei) © Hom(Re,+i, Y r+1 ) 
naturally as Y = Y 1 © Y 2 . Define 

Y+ G Hom(Span({ei, . . . , e r }), Rei) 
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Y l + e l = ei 

Yx e 3 — Yi e j f° r an y 3 > i- 
F 2 + G Hom(Re i+ i, Span({e r+ i, . . . , e i+ i})) 



by 

Similarly, 
is defined to be 

Y 2 + e i+1 = + ei +1 . 

We can prove the following version of Lemma 16.2.91 

Lemma 6.3.1. Let p\ be a representation of N with central character \i realized on 

H = L 2 (Hom(X r , Rei) © Hom(Re {+ i, Y r+1 )) 

as in section 2.1. 
For any 

X G Hom(Span({e r+ i, . . . , ej+i}), Span({ei, . . . , e r })), 

we have 

Pi{X)f{Y)= X i{e Y ? XY >)f{Y). 

Proof. We will write Y as Y\ © Y 2 according to the polarization given in the statement 
of the lemma. Based on the Schrodinger model, if X 6 Hom(y r+ i, X r ), then the action 
of X on the function / at a point Y is multiplication by the character 

Xi(e-^ YX] ) = Xi(e-™). 

If X = X 1 © X 2 G Hom(K r+1 , Rei) © Hom(Me i+ i, X r ) then [X, Y] = -X X Y 2 + Y X X 2 and 
the action of X is by the character 

Xi{e -x lY2+Yl x 2) 

and if X belongs to the center of N then clearly the action will be by the character 

Xi(e X ). 

The statement of the lemma follows by a simple calculation. □ 

One can see that by the duality provided via bilinear form /3, The restriction of a 
representation of rank one to the nilradical of P r is a direct integral of characters which 
correspond to linear operators of the form Y 2 + Y±~, which have rank one (in the usual 
sense) even when the domain is restricted to Rej+i. Proof of the following theorem 
(which shows the equivalence of the two notions of rank) is similar to that of Theorem 

Theorem 6.3.2. Let G = SX; +1 (R). Then the restriction of a pure-rank representation 
ofG of rank k (in the sense of Definition \5. to the abelian nilradical of P r is supported 
on unitary characters of rank k . 

Remark. Note that when I is even, the maximum rank of the unitary characters is |, 
which is the same as the height of the H-tower subgoup of G. However, for odd /, the 
maximum rank of unitary characters exceeds the height of the H-tower group by one. 
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